Skip to content

feat(python): add a Ray Data VortexDatasink - #11478

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:feat/ray-datasink
Open

jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:feat/ray-datasink

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

Ray Data could read Vortex files but had no distributed write path. This adds VortexDatasink, so ds.write_datasink(VortexDatasink(path)) streams each write task to one Vortex file instead of collecting data on the driver or switching formats.

Changes

  • Converts pandas and Arrow blocks to a streaming RecordBatchReader and writes through vx.io.write.
  • Reuses Ray's file-sink path handling, save modes, unique task filenames, and local-filesystem scheduling; includes the Ray 2.48 filename-hook fallback.
  • Adds round-trip, null, append, empty-input, and S3 URL tests plus user documentation.

Verified: 701 passed, 2 skipped for the Python suite; Ray 2.48 and 2.59 datasource suites passed; 317 Sphinx doctests passed. Ruff and ty passed.

API Changes

Adds vortex.ray.datasink.VortexDatasink.

AI assistance

Written with agentic AI assistance; the implementation, tests, and compatibility behavior were reviewed and executed locally.

Stream Arrow batches to one Vortex file per Ray write task, with local and object-store path handling.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant